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ABSTRACT 


Poorly executed and inadequately reported molecular measurement methods are amongst the causes under- 
lying the lack of reproducibility of much biomedical research. Although several high impact factor journals have 
acknowledged their past failure to scrutinise adequately the technical soundness of manuscripts, there is a 

perplexing reluctance to implement basic corrective measures. The reverse transcription real-time quantitative 
PCR (RT-qPCR) is probably the most straightforward measurement technique available for RNA quantification 
and is widely used in research, diagnostic, forensic and biotechnology applications. Despite the impact of the 


minimum information for the publication of quantitative PCR experiments (MIQE) guidelines, which aim to 
improve the robustness and the transparency of reporting of RT-qPCR data, we demonstrate that elementary 
protocol errors, inappropriate data analysis and inadequate reporting continue to be rife and conclude that the 
majority of published RT-qPCR data are likely to represent technical noise. 
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Background 


Biomedical research is supported by immense sums of public 
and private funding, with the NIH alone investing over US$32 
billion annually [1]. This generates tens of thousands of peer- 
reviewed research papers every year and drives the prolifera- 
tion of new hypotheses, guides the direction of fresh research 
efforts, leads to the development of new treatments and so 
underpins further progress. However, the substantial increase 
in the number of scientific publications is also spurred by rea- 
sons other than the desire to communicate results to the sci- 
entific community, not least by the significance of researchers’ 
publication output to their career progression [2], given the 
severe competition for tenured positions [3] and research 
funding [4]. The importance, if not the appropriateness [5], of 
using impact factors to rank the quality of research is well 
established and the most highly cited papers are published by a 
small number of prestigious journals [6]. Necessarily, there are 
hundreds of journals that share the remaining output and 
publish papers that can be equally significant, especially if they 
report technical innovations or report results that are, at the 
time, controversial. Inevitably then, regardless of impact factor, 
a key responsibility of the journal’s editorial team is to ensure 





that there are procedures in place that serve as gatekeepers, 
ensuring that published results are broadly reproducible and, 
ideally, biologically relevant [7]. In theory, this is achieved 
through (i) editorial policies that ensure maximum trans- 
parency through the publication of accurate, comprehensive 
and explicit protocols and (ii) applying rigorous screening 
procedures, most often through the peer-reviewed process [8]. 

In practice, there are significant doubts about the validity of 
many research claims [9] in the context of a flawed research 
infrastructure that encourages disregard for responsible scien- 
tific process, regulation, transparency and reporting [10]. Con- 
fidence in quantitative measurements depends on a number of 
parameters, one of which is reproducibility [11]. Repro- 
ducibility incorporates both biological and technical variability, 
and as long ago as 1949, it was demonstrated that experimental 
test results can vary widely, even when performed by the same 
individual at the same time [12]. Since then, there have been 
numerous publications that highlight the problems of lack of 
reproducibility (reviewed in [13]) and the role journals play in 
failing to enforce their own editorial policies [14]. This, together 
with the fact that credibility and translation are only modestly 
correlated [15], explains why basic research findings are rarely 
adopted into clinical practice [16]. 
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The problem of reproducibility 


Data from a recent online survey run by Nature show that 
respondents believe that there is a significant (52%), or a slight 
(38%), crisis of reproducibility in the peer-reviewed, scientific 
literature, with 65% of respondents experiencing failure to even 
repeat their own results. Curiously, 73% still consider that at 
least half the papers in their field can be trusted [17]. This is 
borne out by the poor reproducibility of biomedical publica- 
tions revealed in two studies [18,19], which has been associated 
with the decline in pharmaceutical research and development 
productivity [20]. Understanding the causes for this predica- 
ment is not straightforward, explanations are complex [21], and 
it has not been possible to identify one specific aspect of the 
published research that is responsible for this quandary [22]. In 
addition to the questionable relevance of many experimental 
models to the real world of human disease [23], the interpre- 
tation of experimental results is hampered by substantial bio- 
logical [24] and technical [25] factors, as well as the deficiencies 
in the validation of experimental designs, [26] avoidable accu- 
mulation of practical errors [27] and inappropriate statistical 
analyses [28]. Together, these are the direct cause for the vari- 
ability of published data that affect reproducibility, possibly 
going some way to explaining why, of the sciences, medicine 
and biology appear to have the lowest reproducibility [29]. 
Undoubtedly, lack of transparency in the Methods section of 
published papers represents a paramount, glaring and self- 
evident explanation, as full disclosure of protocols is essential 
to the integrity and utility of a research publication [30]. 

This subject has been discussed in numerous primary pub- 
lications, reviews and editorials, with a range of causes eval- 
uated in great detail using numerous examples and a range of 
solutions proffered. This includes prestigious and high impact 
factor journals, whose approach comprises assertions such as 
‘Sometimes replication is as important as discovery’ [31] and 
calls for the publication of more detailed results from an 
experiment [32]. Indeed, Nature has acknowledged that edi- 
torial policies have contributed to ‘failures in the reliability 
and reproducibility of published research’ and admitted that 
journals ‘compound’ them by failing to ‘exert sufficient scru- 
tiny’ and not publishing ‘enough information for other 
researchers to assess results properly’ [33], a message reaf- 
firmed by many other Nature titles [34-40]. However, the 
suggested paths to action have remained largely untrodden, 
over the last 5 years peer-reviewed papers of dubious quality 
have continued to be published in low and high impact factor 
journals alike [14,4145] and the scientific literature continues 
to be filled with thousands of papers that report results that 
are at best ambiguous and, at worst, simply wrong [13]. 
Regrettably, to date, the promises from many publishing 


houses to improve the transparency of reporting remain 
unfulfilled. 

This is plainly demonstrated by considering the technical 
quality and completeness of reporting of real-time quantitative 
PCR (qPCR) and reverse transcription qPCR (RT-qPCR) assays, 
the two molecular techniques most widely used today. If the 
entire RT-qPCR workflow is carefully managed from the plan- 
ning, optimisation and validation stages through the actual RT 
and qPCR cycling steps to data analysis and reporting steps, this 
apparently straightforward method can generate rapid, accu- 
rate, sensitive and cost-effective gene expression results [46]. 
However, the simplicity of the protocol and the ease with which 
results can be acquired obscure the pitfall of generating inap- 
propriate or misleading results. Unfortunately, publications 
using both methods continue to be characterised by poor 
experimental design and lack of transparency of reporting of 
technical detail [47], factors recognised as contributors to poorly 
producible research [17], as well as by dubious analysis proce- 
dures [42]. This is despite the publication of the minimum 
information for the publication of quantitative PCR experiments 
(MIQE) guidelines, which define the minimum information 
necessary for evaluating the technical validity of published 
qPCR/RT-qPCR experiments [48], a further checklist for the 
optimisation and validation of qPCR assays [49] and additional 
guidelines for the validation of qualitative qPCR methods [50] 
as well as standards for the application of digital PCR [51]. In 
2013, a review of publications from the Nature publishing 
group concluded that the reporting of essential information 
with regard to RT-qPCR protocols had actually deteriorated 
since the publication of the MIQE guidelines [13]. Whilst the 
guidelines strongly recommended inclusion of essential tech- 
nical information, none of ten papers analysed reported this 
information. This means that the reader was provided with no 
information relating to RNA integrity, RNA purity, RT condi- 
tions or PCR efficiency and all of the publications used an 
inappropriate method of data analysis that was discredited fif- 
teen years ago [52]. We have carried out a repeat analysis on 
twenty recent papers from Nature publishers, which paints 
essentially the same picture (Table 1). Ironically, transparency 
of reporting is negatively correlated with the impact factor of a 
journal [45]. Whilst these analyses relate to technical details and 
reporting inadequacies underlying RT-qPCR experiments, it is 
unlikely that this situation is unique to this single technique. 
Clearly, any change to this current state of affairs requires the 
editors of all journals to take seriously their obligation to ensure 
the technical validity of all papers they publish. 


Techniques used for RNA expression studies 


RNA profiling relies on a number of methods, some relatively 
simple to perform, others much more complex, some limited to 
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Table 1 Continued 


MIQE 


RG validated citation 


No of RG 


PCR conditions PCR efficiency Analysis 


RT replicates RT conditions 


RNA integrity 


Publication Year 


No 


No 


AACq 


No information No information No information 


No information 


2016 No information 


72 


1 


information 


No 


No 


AACq 


No information No information No information 


No information 


2017 No information 


ve 


1 


information 


low throughput and others allowing the parallel processing of 
numerous samples. 


1 PCR is an enabling technology for most of the methods cur- 


rently in use and endpoint PCR analysed using gel elec- 
trophoresis continues to be used today, despite its limited 
quantification capacity. Awareness of variability problems 
associated with PCR has been long-standing, with the first 
report describing inconsistencies with replicate and serial 
specimens evaluated within and between laboratories as 
early as 1992 [53]. The lack of a theoretical understanding of 
the dynamic processes involved in PCR, especially with 
respect to the amplification of nonreproducible and/or 
unexpected amplification products, was also highlighted 
decades ago [54]. These observations and the resulting 
implications are largely disregarded. 

qPCR and RT-qPCR were the first methods to promise rapid 
and easy quantification of nucleic acids and continue to be 
widely used, with a PubMed search for the terms ‘reverse 
transcription real time PCR or RT-qPCR or qRT-PCR’ 
returning nearly 5400 citations in 2016. The need for opti- 
misation of the PCR step as an essential condition influencing 
the quantitative nature of a PCR was stressed from the 
beginning [55], yet qPCR is still referred to as a ‘gold stan- 
dard’ [56,57] and continues to be widely perceived as easy to 
set up, undemanding to carry out, with results readily 
interpreted, even though adoption of that term was chal- 
lenged many years ago [58]. In reality, there are numerous 
critical issues in the workflow that need to be addressed 
before biologically meaningful and trustworthy conclusions 
can be drawn [46] and there have been numerous publica- 
tions questioning the method’s true reproducibility, reliabil- 
ity and consistency (reviewed in [13]). However, 8 years after 
the publication of the MIQE guidelines, many papers 
reporting qPCR data continue to omit the most basic infor- 
mation, as shown in Table 1 and discussed below. 
Microarrays allow much higher throughput than RT-qPCR, 
but due to the increase in complexity and time required to 
perform and analyse the data, this method is declining in 
popularity, with a PubMed search for the term ‘microarray’ 
returning around 1600 citations in 2016. As with RT-qPCR, 
there is evidence of initial lack of robustness of data, with the 
early focus on technical troubleshooting rather than the 
generation of reliable data of scientific significance [59] and 
clinical expression signatures dependent on whatever bioin- 
formatics protocol was being applied [60]. Early cross-plat- 
form concordance has been reported as poor [61], although 
this may be a reflection of variable protocols and standards 
as more recent strict adherence to controlled standard oper- 
ating procedures has resulted in higher intra- and interlabo- 
ratory reproducibility [62]. Again, it is not surprising that 
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there has been emphasis on the need for optimised protocols 
[63], proper quality control measures [64] together with cre- 
ating consensus guidelines for use in clinical diagnostics [65]. 
Given the advent of whole transcriptome shotgun sequenc- 
ing or RNA-seq, it is debatable how much longer microarrays 
will be in use, but they constitute a vast potential resource of 
information and have provided a legacy of published data, 
however reliable, or otherwise, they may be. 

4 Whole transcriptome shotgun sequencing or RNA-seq is the 
technically most complex, expensive and time-consuming of 
all RNA profiling methods, making it less feasible to carry 
out replication or repeat experiments. Initially described as 
being a method where technical variability was too high to 
ignore [66], it is still unclear just how robust and reliable it is, 
although its use is increasing, reflected in the 2700 citations 
recorded by PubMed in 2016. However, the consensus is that 
RNA-seq offers the benefits of superior sensitivity [67] and 
high reproducibility [68], together with the power to identify 
novel transcriptomic features with potential use in clinical 
settings [69]. A multiplatform examination of RNA-seq data 
by the Sequencing Quality Control (SEQC) project found that 
relative, but not absolute, gene expression measurements can 
be measured accurately and reliably across laboratories and 
RNA-seq platforms [70] and that RNA-seq and microarray- 
based models were comparable in clinical endpoint predic- 
tion [71]. Current focus is on data analysis in general, where 
under controlled conditions reproducibility ranges from 60% 
to 93% [72]. Normalisation is a particular aspect where the 
application of different methodologies can generate discor- 
dant results [73-77]. There are also efforts underway to 
generate guidelines that are comparable to MIQE, for every 
stage of the RNA-seq workflow to safeguard the clinical rel- 
evance of the data and so ensuring reliable and robust clinical 
predictions [78]. However, it is worth bearing in mind that 
the reliability of much sequencing data depends on the 
robustness of the RT and PCR steps. 

5 Two additional medium-throughput RNA profiling methods 
are the nCounter Analysis System from NanoString and the 
OpenArray System from Thermo Fisher. The former has the 
advantage of not requiring an RT step, the latter of reducing 
the amount of sample handling required. A comparison of 
mRNA abundance profiling by both technologies and RT- 
qPCR found that results were broadly comparable between 
the three methods. However, whereas NanoString and 
OpenArray results were very well correlated (R = 0-95), RT- 
qPCR data differed significantly from both (R = 0-48 with 
OpenArray and R = 0-55 with NanoString) [79]. The authors 
ascribe this to increased variability in the sample preparation 
caused by the larger amount of technician pipetting and plate 
set-up required for conventional RT-qPCR. Nevertheless, 
these results also demonstrate how results can be discordant 


even though techniques and analyses have been carried out 
in an appropriate manner. It also leaves open the possibility 
that it is the RT-qPCR results that are correct and that those 
obtained by the two other methods are not. This publication 
also serves as an excellent example of the amount of technical 
detail that should be included for the science to be evaluated. 


This brief overview of the major RNA profiling methods 
currently in use illustrates the issues driving the vigorous 
debate about the reliability and biological/clinical relevance of 
many of the data published using RNA-based methods, and 
sheds light on the multifarious reasons underlying the realisa- 
tion that these results are unreliable and in need of revision 
[9,10,16,26,80-87]. 

An accurate assessment of the extent and seriousness of 
irreproducibility has been hampered by the anecdotal nature of 
reports and a lack of quantitative data on specific failures to 
reproduce published biomedical research. This led to the 
establishment of the Reproducibility Project: Cancer Biology. 
The objective of this project was to replicate selected results 
from high-profile preclinical cancer biology research [88]. The 
initial findings from five studies have just been reported and 
indicate that two studies could replicate essential results 
reported by the original authors, one could not and two were 
uninterpretable because of technical issues. Interestingly, one of 
the replicated results [89] confirmed selective downregulation 
of MYC transcription by reverse transcription (RT) quantitative 
real-time polymerase chain reaction (qPCR) [90]. The original 
experiment compared MYC mRNA levels in control cells and 
cells treated for 1 or 8 h with a small molecule inhibitor and 
recorded a downregulation in treated cells to 6-9% and 8-8% of 
control cells. The replicate study recorded a downregulation to 
14-6% and 12:6%, which is indeed similar but nevertheless 
amounts to a twofold difference. This is important because the 
changes in RNA levels reported by many publications using 
RT-qPCR data tend to be in that range, typically between 1-5- 
and fivefold. 

A consequence of the superficially simple process of the RT- 
qPCR workflow is that on the one hand, many untrained and 
unskilled operators have adopted the technique. On the other 
hand, knowing where to look allows an experienced reader of a 
publication to identify problem areas and ascertain whether the 
reported results are likely to be real. Below follows a summary 
of the methodological issues that impede consistent interpre- 
tation of results, together with explicit examples that illustrate 
graphically why reported results are so frequently unreliable. 


Pre-analysis 


Relevant quality control measures are an essential preparation 
preceding any molecular biology-based experiment, including 
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ensuring reliable qPCR assays. There are numerous inhibitors 
that can be copurified during sample preparation and there are 
a variety of physical and chemical factors that degrade nucleic 
acids during the preparation process or subsequent storage. 
Hence, it is important that care is taken to determine whether 
there is a problem with purity and integrity and, for quantita- 
tive applications, this should be measurable and quantifiable 
[91]. In practice, however, most publications do not even 
mention sample purity, as measured by the absence of inhibi- 
tion [45]. Yet the problems associated with inhibition have been 
well known for a long time [92] and their consequences are 
obvious. Even carrying out a general assay aimed at detecting 
inhibition [93] is probably not sufficient, as different PCRs are 
differentially susceptible to inhibition by substances copurified 
in nucleic acid extracts [94]. Although this has tremendous 
implications for microarray- and sequencing-based methods as 
well as molecular diagnostic applications, the phenomenon has 
been inadequately explored and remains ignored or at least 
unreported. 

In principle, reliable quantification of RNA requires reason- 
ably intact RNA [95]; the question is what constitutes ‘reason- 
ably’ intact and how best to measure it. For microarray analysis, 
it has been suggested that comparisons are valid if they are 
done using samples of comparable RNA integrity [96] and a 
similar finding has been reported for RT-qPCR [97]. However, 
one needs to ask the question of how significant or correct 
results obtained from low-quality RNA are, especially if they 
are discordant from those obtained using high-quality RNA. 
Whilst it may be technically feasible to obtain data from such 
RNA, a demonstration of biological or clinical relevance is often 
missing. Since the original demonstration that assessing RNA 
quality by gel-based analysis of 18S and 28S rRNA is not suf- 
ficiently rigorous for quantitative applications [98], several 
methods have become available to assess RNA integrity. Some 
use a microcapillary electrophoretic separation system to pro- 
vide a general assessment of overall RNA integrity through an 
RNA integrity number (RIN) [99], others measure mRNA 
degradation by comparing the amplification of different length 
amplicons of specific reference genes [100] or by evaluating the 
relative PCR amplification of 5’ and 3’ regions of cDNA 
obtained by oligo-dT priming [58]. Unfortunately, the majority 
of qPCR-based publications do not use or report data from such 
analyses and, at best, report spectrophotometric analysis data 
that are quite irrelevant for quantitative methods [101]. 

One suggestion is that total RNA with a RIN value of above 
five can be classified as good quality and above eight as perfect 
quality [95]. However, this is not an ideal categorisation as 
degradation is similar to inhibition in that it is not constant or a 
steady process but affects transcripts in different ways [102]. A 
serious consequence of this inconsistency is that comparisons of 
samples that differ in their purity or degree of degradation are 


likely to show differences where there are none or alternatively, 
mask any actual differences. This inherent instability com- 
pounds any inconsistencies arising from the intrinsic biological 
variability [58] and directly affects gene expression results and 
so distorts the clinical relevance of prognostic markers and risk 
classification [103]. Tellingly, even the adoption of rigorous 
quality criteria for RNA integrity assessment does not always 
prevent laboratory-specific effects leading to significant vari- 
ability, although validated and standardised RT-qPCR assays 
can achieve accurate results [104]. 

It is worth noting that these considerations also apply to any 
qPCR experiments targeting DNA, where the choice of extrac- 
tion method and the techniques used to quantify the extracted 
DNA, as well as the presence of inhibitors combine to result in 
measurement uncertainty [105]. 


The RT step 


A scrutiny of the history of RT-PCR uncovers the unfortunate 
truth that the fundamental issues dogging the reliability of 
today’s experiments were identified at an early stage, were 
constantly expounded and brought to the research commu- 
nity’s attention, but continue to be disregarded by most. The 
potential of combining the RT and PCR steps to detect RNA 
targets was realised soon after the reporting of the first PCR 
experiment [106], with its use as a quantitative assay suggested 
soon afterwards [107]. However, it very quickly became clear 
that quantification of RNA was not quite as straightforward as 
applying the same procedures used for quantification of DNA 
as, for example, reliable reverse transcription of RNA for 
amplification by PCR depends on RNA concentration and 
priming methods [108] as well as the avoidance of extensive 
secondary structures at the primer binding sites [109,110]. 
Awkwardly, different RT enzymes are differentially affected by 
these secondary structures [111]. 

Misgivings with regard to the capacity of RT-PCR to distin- 
guish small fold changes [112] were soon followed by reports of 
problems with lack of reproducibility of low copy number 
target detection [113], inhibition of the PCR by the RT enzyme 
[114], primer-dependent variability resulting in marked varia- 
tion of RNA copy number [115] and a realisation that inter- 
laboratory differences in the protocols used for sample 
preparation, RNA extraction and reverse transcription were 
responsible for discordance in published data involving end- 
point RT-PCR [116]. The need for detailed quality assurance 
[117] and standardisation [118] at every step was highlighted 
well before the era of routine RT-qPCR usage, as were the 
challenges posed by the significant variability in RT and PCR 
efficiencies [119]. 

The coupling of RT with qPCR as a method for real-time data 
acquisition with RNA targets held out the promise of a 
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quantitative, as well as sensitive, accurate and high-throughput 
method [120] that might be more robust and reliable. However, 
substituting RT-PCR with RT-qPCR, in itself, does not control 
for RT variability [119]. Indeed, a careful reading of that first 
RT-qPCR paper reveals that the mean quantification cycles (Cq) 
of two RT-qPCR assays repeated on three separate days were 
reported as ranging from 29-3 to 29-6 and 17-4 to 18-3, respec- 
tively, that is the variability corresponded to 1-2- and 1-9-fold 
changes. Furthermore, there can be a priming-dependent lack 
of linearity of the RT step [121] and oligo-dT priming can lead 
to the generation of truncated cDNAs [122]. RNA yield at low 
template concentrations depends on the choice of RT enzyme 
[123] and accurate quantification of the amount of input RNA 
can be affected by variable amounts of contaminating DNA left 
after DNase treatment in the supposedly ‘pure’ RNA template 
[123]. What is more, the accuracy of RT-qPCR results is signif- 
icantly affected by operator variability [123], which, by defini- 
tion, introduces variation within and between laboratories. 

The conclusive demonstration of the true variability of the RT 
step was provided by two key publications which ascertained 
that experimental variation in RT-qPCR is largely determined 
by the RT step. The first measured the properties of the RT 
reaction for five mRNA targets using different cDNA priming 
strategies [124]. Results not only confirmed the connection 
between priming strategy and both RT efficiency and RNA 
concentration, but also demonstrated that this dependency was 
target-specific, with the measured expression ratios of two 
genes varying by as much as 24-fold under different reaction 
conditions. Not surprisingly, the authors emphasised the need 
to optimise the RT step, carrying out at least duplicate RT 
reactions, adjusting the total RNA concentrations and always 
using the same reaction conditions. The second paper extended 
these findings to a comparison of the results obtained using 
eight different RT enzymes [125]. Disconcertingly, RT yields 
varied by up to 100-fold, dependent on which RT was used, 
and variation was also target-dependent. Unfortunately, the 
authors did not point out an obvious conclusion from both sets 
of results, namely that a quantitative technique generating 
inconsistent results with different reagents is neither effective 
nor useful. It is also worth reflecting on the fact that despite the 
central importance of these publications for understanding the 
properties of the RT reaction in mRNA quantification, their 
impact has been rather limited as citations are measured in the 
low hundreds compared to many thousands for key technical 
qPCR papers. More recently, these results were extended to 
demonstrate that the technical variability of the RT step is 
greater than the fold changes reported by many, if not most 
publications using RT-qPCR [126], again without much reso- 
nance in the research community. 

Taken together, the conclusion inferred from these findings 
is that the RT step is intrinsically variable as it is carried out 


by different individuals that use variable amounts of RNA, a 
variety of cDNA priming methods and different RT enzymes 
together with a myriad of protocols. Awkwardly, the real-life 
implications of these findings are stark: a necessary conse- 
quence is that that RT-qPCR results reported by different 
laboratories are likely to be discordant and it is not an easy 
matter to ascertain which results are real and which ones are 
noise, especially if there is inadequate information on assay 
design parameters and experimental protocols. Clearly, this 
deduction is valid for any method that relies on the RT step, 
for example digital RT-PCR (RT-dPCR), where results are also 
both RT- and target-dependent [127], and RNA-seq, where 
technical variability has been found too high to ignore and has 
resulted in inconsistent detection of exons at low levels of 
coverage [66]. 


The PCR step 


This step, usually described as a simple enzymatic reaction, is 
generally accepted to be the least troublesome and most reliable 
segment of a RT-qPCR assay, assuming that a few important 
quality criteria have been addressed. These include suitable 
assay design, empirical optimisation and validation of primers 
and amplicons and the absence of inhibitors in the template 
being amplified. Indeed, under those conditions PCR results 
are so reproducible that a traditional recommendation is to 
carry out replicates of the RT step and that it is acceptable to 
dispense with performing multiple qPCR replicates on each RT 
sample [128]. Alas, the robustness of the PCR step is not quite 
what it seems and there are several reasons for this. 


1 Many published PCR assays are either inadequately 
designed and are unlikely to amplify only the intended target 
or the published information is incorrect, resulting in the 
publication of wrong primer or probe sequences. An example 
of the former is a qPCR assay for a truncated mRNA splice 
variant reported as a biomarker for predicting future metas- 
tasis in human metastatic colon, breast and hepatocellular 
cancers [129]. Analysis of the forward primer shows that it 
has sequence identity with 16 of 20 nucleotides present in the 
nontruncated mRNA, 15 of which are at the 3’ end, likely to 
result in effective nonspecific priming. An example of the 
latter is the reporting of a hydrolysis probe sequence that 
targets a region outside the PCR amplicon defined by the 
primers [130]. The surprise is that such basic mistakes can 
pass through the peer review system, as inadequate assays 
are easy to spot by a simple examination of the oligonu- 
cleotide sequences and assay location. These observations 
were only possible because the authors made their assay 
sequences available and therefore lend weight to the original 
MIQE [48] requirement for assay sequences to be published 
in full. 


762 © 2017 Stichting European Society for Clinical Investigation Journal Foundation 


LACK OF REPRODUCIBILITY 





2 Optimal PCR performance is very much dependent upon 
appropriate template preparation methods [131] and there 
have now been more than 100 publications that have com- 
pared the performance of commercial extraction kits, and 
many more that of home-brew methods (recently reviewed in 
[105]. All of these comparisons show that it is important to 
choose extraction methods or commercial extraction kits with 
care, as they have a significant impact on the variability of 
subsequent PCR results. This strongly suggests that anyone 
contemplating carrying out qPCR assays should not just 
compare qPCR chemistries, reagents and instruments prior 
to settling on the optimal ones but also investigate nucleic 
extraction procedures, report the results and explain the final 
choice of extraction method. 

3 The presumption that qPCR assays can reproducibly resolve 
small fold changes is misplaced. The specific manufacturer's 
claim of a twofold discriminatory power at 10 000 target 
copies is generalised to suggest that qPCR can always sepa- 
rate twofold differences. The concept is based upon standard 
instrument validation tests where B-actin copy numbers were 
measured in 36 replicates of 10 000 and 20 000 genome 
equivalents. As B-actin has multiple pseudogenes, the actual 
copy numbers measured may well have been greater than the 
genome equivalent. Therefore, it is inappropriate to extrap- 
olate the instrument validation test to an assumption that all 
differences of twofold can be detected accurately, especially 
when running few, if any, replicates. As Fig. 1a and b shows, 
qPCR can indeed discriminate twofold differences and can 
do so reproducibly. However, at low target concentrations 
this is no longer possible, unless a large number of replicates 
are used and even then, there is a significant overlap of 
recorded Cqs. Furthermore, as Fig. 1c shows, it is clearly not 
possible to discriminate a 1-5-fold difference, something 
which is never claimed by manufacturers yet frequently 
reported. Just to quote two examples, Fig. 5 of a one paper 
reports 19 gene expression changes, of which 18 are < 1-5- 
fold [132] and Fig. 1 of another publication claims to be able 
to show 1-25-fold differences [133]. 

4 It is not much appreciated that all master mixes are not the 
same and that the use of modified polymerases, variable 
buffer additives and different Mg2+ concentrations can 
generate sufficiently variable results when different master 
mixes are used. Commercial master mixes vary in their 
ability to perform reliably across all instrument platforms 
[134], and indeed, some master mix/thermal cycler combi- 
nations are inadvisable [135]. Furthermore, the PCR step is 
known to introduce base composition bias in high-through- 
put sequencing libraries and DNA polymerase has been 
pinpointed as its principal cause [136], requiring modifica- 
tion and optimisation of standard amplification conditions. 
The extent and type of bias vary with different commercial 


master mixes, so much so that the avoidance of some master 
mixes may be required [137]. One reason for this bias is that 
sequences at the 3’ end of the primer as well as in the PCR 
amplicon itself appear to affect priming efficiencies of DNA 
polymerases, with different commercial enzymes having 
different preferences [138]. Hence, it is essential to under- 
stand that whilst replicates using the same master mix are 
very tight, repeat experiments using different master mixes 
can vary by at least and much as many of the reported fold 
differences. This is of particular relevance in the instances 
where Cq values only or Cq values relative to a single ref- 
erence gene, a comparison referred to as ACq, are recorded 
and then published as fold differences. Figure 2 compares 
the expression of eight genes relative to a single reference 
gene after the amplification of the same cDNA with four 
master mixes. The variability recorded for each master mix, 
as evidenced by the error bars in Fig. 2b, is small, but con- 
sidering that all reactions were carried out at the same time 
with the same instrument, the differences observed between 
master mixes are instructive. The variation in ACgs ranges 
from 3-4 to 5-1, which translates into fold differences from 
around 10- to about 30-fold (Fig. 2c). Without knowing the 
amplification efficiencies, it is impossible to resolve whether 
these differences could be caused by the three assays having 
different master mix-dependent priming or amplification 
efficiencies. 


However, most RNA expression experiments compare the 
relative expression of markers between two samples, usually a 
test and a reference sample, calculating a fold difference, and 
the results shown in Fig. 3 show the technical noise expected 
from this type of comparison. The experiment involved using 
master mixes MM1 and MM2 to amplify cDNA prepared from 
two breast cancer RNA samples. The amplification plots for 
both master mixes are similar and the chosen reference gene, 
GAPDH, is expressed at similar levels in both RNA samples 
(Fig. 3a). However, results for genes of interest vary between 
two- and fivefold (Fig. 3b). Bearing in mind that these experi- 
ments were carried out by one researcher using RNA extracted 
by the same method, the same reagents and instruments, these 
fold differences constitute a minimum and could well increase 
if carried out by a different laboratory. Furthermore, the use of 
different primers may lead to smaller or larger differences. 


5 RT-qPCR experiments can be carried out using separate RT 
reaction and PCR (two-step assays) or a combined reaction 
(one-step assay). The former tends to use oligo-dT and/or 
random primers, which provides more flexibility and enables 
the design of PCR-optimised target-specific primers. The 
latter minimises the hands-on time required to complete the 
assay and may be more sensitive depending on what assay is 
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Figure 1 Resolution of a qPCR assay. (a) Amplification plots showing results from qPCRs carried out on genomic DNA, which was 


diluted twofold and subjected to a qPCR assay, with 48 replicates for each sample (neat sample is shown in red and diluted sample 
in blue). The observed ACq was 1-22, compared to a theoretical ACq of 1. (b) The assay was repeated on a different day with a fresh 
dilution of the sample and the ACq was 1-06. (c) The original sample was first diluted at a ratio of 1 in 100 and then again twofold. 
This time, additional qPCR assays were carried out using 1-5 times the amount of starting material (green amplification plots) from 
the twofold diluted sample, with 32 replicates for each sample. (d) Comparison of copy numbers calculated from a standard curve 
for the assay confirms the lack of resolution at this template concentration. 


used. Comparisons between the two methods have con- 
cluded that results are broadly similar [139,140]. As always, it 
is dangerous to generalise, as this equivalence is not always 
observed. Figure 4 shows an example of the two approaches 
giving different results. Four mRNA targets were targeted in 
two RNA samples using either a two- or a one-step protocol. 
It is immediately apparent that the VDR results are signifi- 
cantly different for the two types of assay, with the one-step 
assay results suggesting either very low expression or primer 
dimer amplification. Four repeats of both assays gave the 
same result, indicating a problem with the robustness of the 
VDR data. An analysis of the secondary structure of the RNA 


(Fig. 4b) reveals extremely strong stem structures for all 
predictions throughout the template. One possible explana- 
tion therefore is that the target-specific primer used for the 
one-step assay could bind only inefficiently, whereas the use 
of random priming in the two-step assay allowed the RT to 
read through the sites subsequently used for PCR priming. 
However, both IGF1 and MMP3 primers also target sec- 
ondary structure-rich RNA, so this cannot be the whole 
explanation. Furthermore, a comparison of fold changes 
reveals additional variability between the two methods 
(Fig. 4c). Whilst both generate concordant results for IGF1, 
they are discordant for MMP3. Without further detailed 
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Figure 2 Master mix-dependent variability in cDNA quantification. (a) RNA extracted from a cell line with a RIN value of 10 was 
reverse-transcribed and nine targets were amplified with four master mixes (MM1-MM4) under identical, optimal conditions. (b) 
The expression of each target relative to that of GAPDH demonstrates the variation recorded in the ACq for each marker. (c) Plot of 
the maximum differences in ACq values and their translation into fold differences. 


examination, it is impossible to ascertain why this is so. 
However, it reinforces the importance of careful primer 
design if the target is RNA. 

6 One might assume that quality control of PCR reagents had 
been perfected after more than thirty years of the technique 
being at the forefront of molecular biology research. An early 
observation of maintaining PCR reagents at temperatures 
above the annealing temperature of the primers resulted in 
more specific amplification products [141] led the develop- 
ment of hot-start polymerases [142,143], which are now in 
common use for all varieties of PCR. Their use is especially 
important for high-throughput applications, where PCR 
reagents and templates are dispensed hours ahead of run- 
ning the eventual PCR, giving any enzyme retaining poly- 
merisation capacity plenty of time to generate spurious 
amplification products that could affect the sensitivity, 
specificity and reproducibility of the results. Unfortunately, 
many commercial hot-start polymerases appear not to have 


been tested by the suppliers for polymerase activity prior to 
thermal activation, as a recent report shows that 12 of 17 
polymerases tested showed such activity when analysed by 
gel electrophoresis [144]. 


7 In the aforementioned survey [17], just under 60% of 


respondents posited that poor analysis was an important 
contributor to irreproducible research. Yet even for a method 
supposedly as simple as qPCR, there are numerous ways to 
analyse data. The need to address measurement uncertainty 
contributions made during the data analysis procedure was 
identified early on as important to minimise potential 
uncertainty in results due to subjective judgements [145]. The 
most commonly used method for comparing RNA transcript 
levels involves relative quantification against one or, prefer- 
ably more, reference genes using the AACq method [146]. 
Although this approach has long been known to absolutely 
require knowledge of the PCR efficiencies of genes of interest 
and reference genes [147], repeated surveys show that most 
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Figure 3 Technical noise of an RT-qPCR experiment. (a) Amplification plots of four targets amplified in duplicate from cDNAs 
prepared from two different high-quality RNAs (one shown in blue, the other in red) with RIN = 10 with MM1. (b) Amplification plots 
of the same targets amplified in duplicate from the same cDNAs (one shown in purple and the other in brown) with MM2. (c) The 


relative fold changes in expression of the three genes of interest (GOI) relative to that of GAPDH between the two samples is plotted 
(MM1 is blue and MM2 is pink). 


researchers very rarely measure efficiency and assume it to precision, especially if they bypass assumptions regarding 
be 100% [13,45,148], making any conclusions based on such PCR efficiency [155]. Indeed, the question of how to calculate 
experiments doubtful. This is still the case today, as the PCR efficiency is another one of those many unsolved ques- 
analysis in Table 1 reveals. About 8-10 years ago, a flurry of tions with respect to qPCR analysis, not least because 
reports appeared, all attempting to introduce flexible, recorded PCR efficiencies vary significantly across different 
understandable and cost-effective data analysis and man- instruments [156] and the performance of the different anal- 
agement tools [149]. These include qPCR-DAMS [150], ysis tools cannot be easily compared as most use their own 
Q-Anal [151], qpcR [152], QPCR [153] and qBase [154]. file format [157], raising the question of whether and how 
However, the various methods differ significantly in their raw data should be submitted [158]. The standard curve 
performance, and whilst some generate reproducible and approach is the most popular [159], but alternatives that 
sensitive data, they do not always quantify cDNA with estimate PCR efficiency based on the analysis of individual 
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Figure 4 Comparison of one-step and two-step RT-qPCR. (a) Replicate Cq values recorded for one-step or two-step RT-qPCR 
assays carried out using the same RNA and PCR conditions and showing the lack of amplification for VDR with the one-step 
approach. (b) Secondary structure of VDR mRNA. (c) Comparison of the fold differences in expression of the two genes of interest 
(GOI) relative to that of GAPDH between two 1-step (Dark and light blue) and two 2-step (red and pink) protocols shows that results 


are concordant for one target but not the other. 


amplification curves when using SYBR Green I detection 
chemistry claim to model PCR amplification more effectively 
[160-163]. 


These examples demonstrate the variability that can arise 
during the qPCR step and show that this is due to (i) inherent 
limitations of the ability of thermal cyclers to resolve small fold 
changes, especially at low template concentrations, (ii) varia- 
tions between the master mixes caused by the use of different 
polymerases and additives to the buffers, (iv) differences 
between the performance of one-step and two-step RT-qPCR 
master mixes and (iii) insufficient attention by the manufac- 
turers to quality control, resulting in partially active enzymes 
that could impinge on specificity and sensitivity. Notably, 
inappropriate RT-qPCR results are not simply ‘false negative’ 
or ‘false positive’, but are recorded as variable degrees or 
directions of relative changes in expression, expressed as fold 
differences. However, false positives are a problem if primers 
are not sufficiently specific, hence the need to validate them 


empirically and the strong recommendation to report their 
sequences [164,165]. 


Examples of poor practice 


Two recent publications describe attempts to identify gene 
expression patterns associated with major depression disorder 
(MDD)-related behaviour or its treatment. The first used 
microarrays to identify differentially expressed transcripts in 
two animal models [166] and measured the relative expression 
of their human orthologs in the peripheral blood of patients 
diagnosed with MDD and control subjects using RT-qPCR. The 
authors concluded that they had established a clinically valid 
diagnostic panel of blood transcripts genes that can differenti- 
ate early-onset MDD from controls and MDD with or without 
comorbid anxiety. This study was extended in a follow-up 
study with a new cohort of patients [167], which examined the 
change in gene expression in patients with MDD prior to and in 
response to cognitive behaviour therapy (CBT) by RT-qPCR. As 
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the conclusions of the two studies are based on interpretation of 
RT-qPCR data, it is germane to scrutinise both publications for 
technical validity and analytical soundness. 

As is usual, there is little transparency of reporting of technical 
information: (i) the first paper [166] does not mention either RNA 
integrity or purity assessment, the second refers to spectrometric 
analysis but provides no information on sample quality, (ii) there 
is inadequate reporting of RT conditions, (iii) a single reference 
gene is used with no evidence that this is expressed stably across 
all conditions, (iv) there are mistakes in the published primer 
sequences, (v) primers do not detect all splice variants, and (vi) 
one assay is nonexon spanning, but no controls are provided to 
show the absence of contaminating genomic DNA. 

However, the most troubling issue is concerned with the 
analysis and interpretation of the data. Fortunately, the authors 
have included some limited analysis data in each publication. 
These have been collated in Table 2. In detail, we note the fol- 
lowing errors: 


1 The authors record ACq values between the genes of interest 
and the reference gene, but there is no evidence that they 


evaluated PCR efficiency. As discussed previously, this is 
enormously concerning, especially in the light of the minute 
differences in expression being reported. 

2 A comparison of ACqs from the two studies reveals a striking 
difference between the 2012 and 2014 studies, with all tran- 
scripts recording higher values in the later paper compared 
with the earlier one (Table 2). The average Cq difference for all 
markers in the MDD groups was 4-81 (or 28-fold) and 5-5 (or 45- 
fold) in the control groups. However, this masks large differ- 
ences between the individual markers, with FAM46A record- 
ing the same results and CADM1/IGSF4A differing by 
sevenfold. As the methods sections of both studies are virtually 
the same, these differences strongly suggest a technical dif- 
ference between the studies. However, as there is inadequate 
information with regard to the protocols used, it is impossible 
to discern whether there were potentially significant differ- 
ences in RNA quality, conversion to cDNA, qPCR efficiency or 
analysis procedures or a combination of these parameters. 

3 The AACq method of calculation has been applied incor- 
rectly, as the authors report negative fold changes and it is 


Table 2 Genes and ACq values extracted from two publications analysing MDD [166,167] 


Gene 2012 MDD 2014 MDD Cq difference 2012 ND 2014 ND Cq difference 
ATP11C 15-26 19-22 3-96 14-92 19-20 4-28 
CD59 14-65 19-98 5-33 13-98 19-89 5-91 
FAM46A 15-31 19-65 4.34 15-62 19-96 4.34 
CADM1/IGSF4A 18-42 22.22 3-80 16-50 23-05 6.55 
MAF 18-12 22-00 3-88 17-70 22-08 4.38 
MARKS 13-52 16-85 3.33 13-29 17-41 4-12 
NAGA 13-69 17-63 3-94 13-90 17-85 3-95 
RAPH1 17-11 21-73 4-62 15-70 22-80 7-10 
TLR7 15-04 19-53 4-49 15-43 19-89 4-46 
ZNF291 16-17 23.55 7:38 16-16 23-20 7-04 
AMFR 13-98 19-28 5-30 13-57 19-27 5-70 
CDR2 15-55 20-60 5-05 14-88 20-49 5-61 
CMAS 17-32 22-41 5-09 15-53 22.98 7-45 
DGKA 10-31 16-69 6-38 11-24 17-41 6-17 
PSME1 11-66 17-17 5-51 11-20 17-55 6-35 
PTP4A3 15-80 20:79 4-99 14.95 21-03 6-08 
SLC4A1 12-69 17-08 4-39 13-19 17-22 4-03 
Average 4-81 5-50 
SD 1-01 1-20 


The two studies use different cohorts of patients and controls but have similar clinical characteristics. MDD, major depression disorder. 
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impossible to produce a negative value from the Livak 
algorithm (244C4) The actual AACq values between MDD 
and control groups for each marker are very small in both 
studies, suggesting minimal differences in expression 
between the two groups. For example, the AACq values for 
ATP11C were 0-79 and 0-99 in the earlier and later reports, 
respectively. The correct changes have been recalculated 
for each data set and are summarised in Fig. 5. None of 
the data show any genes with significant fold changes, 
beyond that which could be explained by experimental 
variation. In fact, RAPH1, the gene highlighted as poten- 
tially diagnostic and for which a patent has been applied 
for shows a change in direction of expression between the 
two studies. 


Guidance 


There are a number of obvious steps that researchers car- 
rying out RT-qPCR experiments and reviewers or readers of 
a publication can take to determine how reliable any results 
and subsequently reported data are likely to be. The aim is 
to ensure the experimental design and workflow controls as 
much as possible for biological variability, caused by 
inherent differences between individual organisms, tissues 
or cells and technical variability originating from experi- 
mental elements. The former is best addressed through the 
use of three or so biological replicates per experimental 
sample whenever feasible, the latter using three technical 
replicates per reaction. 


1 In silico verification of primer and probe sequences. 
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Figure 5 Recalculated fold changes for the targets claimed to 
show significant differences between patients with MDD and 
normal controls. The black bars show the values recorded by 
the earlier publication [166]. 


1.1 As the specificity and efficiency of any PCR assay are 
critically dependent on the primers and probes, if used, 
these need to be carefully checked. Unfortunately, there 
are frequent mistakes, including simple typographical 
errors, confusion about the oligonucleotide sequence 
orientation, primers that lack specificity, bind to multi- 
ple targets, have 3’ ends that target SNP sites, are not at 
all or insufficiently intron spanning or probes that bind 
to targets outside the PCR amplicon. For one tube 
combined RT-qPCR assays, it is also important to 
establish that the cDNA primers do not target strong 
secondary structures of the RNA, as this results in 
variable, low efficient priming by the reverse transcrip- 
tase. 


2 Establish the purity (absence of inhibitors) and determine the 
quality (integrity) of nucleic acid samples following their 
extraction. 


2.1 The absence of inhibition can be tested either by 
diluting a sample and comparing the Cqs or by using a 
reporter such as the SPUD assay. The former method 
requires more sample and reagents, but allows an 
assessment of any potential inhibition on the targets of 
interest. The latter is less labour-intensive but may miss 
any target-specific inhibition. Sample quality is best 
measured using a microcapillary system or a 3'-5' 
assay. Reporting an OD 60.289 ratio is meaningless for 
experiments aiming to quantify accurately small dif- 
ferences in RNA concentrations. Inadequate reporting 
means that it is never clear whether authors actually 
check for the presence of inhibitors and ascertain an 
RNA sample's integrity or just do not report their 
results. 


3 The RT step should be carried out in triplicate if at all pos- 
sible, whereas it is sufficient to run a single qPCR for each RT 
reaction. 


3.1 Most reported fold changes are relatively small and the 
variability of the RT step, especially when using low 
concentrations of RNA, has been demonstrated to be 
greater than many of the reported fold changes. Yet 
there are virtually no publications that follow this rec- 
ommendation. The qPCR step, on the other hand, is 
highly reproducible, if the same master mix is used by a 
skilful operator and so multiple reactions are less 
informative. Alas, a change of master mix can generate 
different results, especially if the expression changes are 
marginal. 
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4 The efficiency of each individual PCR assay must be ascer- 
tained. 


4.1 The validity of a quantitative result cannot be determined 
reliably and reproducibly without knowing whether the 
PCR assays were efficient and incorporating any vari- 
abilities into fold change calculations. PCR efficiency can 
be calculated from standard curves or using published, 
often free-to-use, software. Most qPCR instruments pro- 
vide an option to calculate PCR efficiencies automati- 
cally. The use of the AACq method in the absence of 
information on PCR efficiency is always a clear indicator 
that the data are suspect. If researchers were to address 
only this one element of the RT-qPCR workflow, this 
would arguably result in the biggest improvement to the 
accuracy of gene expression analyses. 


5 The use of multiple, stable and validated reference genes is 
essential for RT-qPCR data normalisation as it removes 
variation due to inherent technical or experimentally induced 
variation. 


5.1 Normalisation against a single, unvalidated reference 
gene to report small fold changes has long been known 
to invalidate quantitative results and results in erro- 
neous normalisation up to threefold and 6-4-fold in 25% 
and 10% of the cases, respectively, with sporadic cases 
showing error values above 20% [52]. In contrast, most 
reports deal with fold changes that range from 1-5-fold 
to about eightfold, all of which are definitely included in 
this category. Importantly, depending on the amount of 
target being quantified, fold changes less than three- to 
fivefold are likely to be unreliable. 


6 Cq data should be submitted as supplementary information, 
either in a spreadsheet or directly exported in RDML format. 


Conclusions 


The ostensible simplicity of RT-qPCR conceals a complexity 
that extends to every step of the workflow and there is a long 
history of publications analysing, untangling and clarifying 
the issues that must be confronted to ensure the appropri- 
ateness of assay design, robustness of the protocol and finally 
the suitability of analysis methods. On the other hand, there 
are numerous publications that have been highlighting seri- 
ous concerns with the relevance, reproducibility and practi- 
cability of biomedical research. Considering the key enabling 
roles of the RT and PCR steps in other molecular technolo- 
gies, it is reasonable to put the two issues together and con- 
clude that there is a link between the two. It is shocking that 


despite the easy availability of this information, the quality of 
execution of a technique that has been in use for 24 years 
remains at such a low standard. Either its users are ignorant 
of the complications involved in ensuring reproducibility or 
they deliberately ignore the wealth of advice and knowingly 
publish inadequate results. It is also clear that the availability 
of guidelines has done very little to improve the quality of 
published data based on the use of RT-qPCR and begs the 
question that if a technique where inadequacies are easily 
detected is as badly executed as this, what must the situation 
be like for the much more complex technologies based on it. 
The effects of this negligence by the research community to 
heed the many warnings that have been issued over the years 
is enhanced by the failure shown by the editors of most 
journals to implement their own instructions for authors, 
despite publishing editorials and seemingly supporting ini- 
tiatives aimed at improving this situation. This unwillingness 
to take remedial action in general and for RT-qPCR in par- 
ticular means that much taxpayers’ money and donors’ gen- 
erosity continue to be wasted. 
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